Searching trees: live time-lapse cell-cycle progression modeling and analysis

ABSTRACT

A method of tracking a cell through a plurality of images includes selecting the cell in at least one image obtained at a first time, generating a track of the cell through a plurality of images, including the at least one image, obtained at different times using a backward tracking, and generating a cell tree lineage of the cell using the track.

BACKGROUND

The present disclosure relates to methods for producing temporal celltrajectories using accurate cell segmentation.

Methods of understanding embryonic development, such as in tracking celllineages, are an important aspect of developmental biology. The abilityto track individual cells changes (e.g., cell morphological changes)during development has existed for some time. However, the ability totrack cells over longer time periods or when the cells migrate throughthe body is not currently known. For example, the ability to track cellsover time is hampered by the large amounts of time-lapse image dataproduced by live cell imaging, which is typically more information thancan be digested by a human observer. Moreover, interpreting the imagedata, e.g., identifying different cells having with different shapes andvectors, and which are disposed in densely-packed groups, is achallenging task.

BRIEF SUMMARY

According to an exemplary embodiments of the present invention, a methodof tracking a cell through a plurality of images includes selecting thecell in at least one image obtained at a first time, generating a trackof the cell through a plurality of images, including the at least oneimage, obtained at different times using a backward tracking, andgenerating a cell tree lineage of the cell using the track.

According to at least one embodiment of the present invention, a methodof tracking a cell through a plurality of images includes selecting thecell in at least one image obtained at a first time, generating a trackof the cell through a plurality of images, including the at least oneimage, obtained at different times using a backward tracking tracing thecell back in time until detecting a mitosis event of the cell, modelinga cell cycle duration of the cell using a time of the mitosis event,modeling a rate of mitosis of the cell using the backward tracking,labeling, simultaneously, a plurality of cells, including the cell, withdifferent identities in the plurality of images, and generating a celltree lineage of the plurality of cells using the track.

As used herein, “facilitating” an action includes performing the action,making the action easier, helping to carry the action out, or causingthe action to be performed. Thus, by way of example and not limitation,instructions executing on one processor might facilitate an actioncarried out by instructions executing on a remote processor, by sendingappropriate data or commands to cause or aid the action to be performed.For the avoidance of doubt, where an actor facilitates an action byother than performing the action, the action is nevertheless performedby some entity or combination of entities.

One or more embodiments of the invention or elements thereof can beimplemented in the form of a computer program product including acomputer readable storage medium with computer usable program code forperforming the method steps indicated. Furthermore, one or moreembodiments of the invention or elements thereof can be implemented inthe form of a system (or apparatus) including a memory, and at least oneprocessor that is coupled to the memory and operative to performexemplary method steps. Yet further, in another aspect, one or moreembodiments of the invention or elements thereof can be implemented inthe form of means for carrying out one or more of the method stepsdescribed herein; the means can include (i) hardware module(s), (ii)software module(s) stored in a computer readable storage medium (ormultiple such media) and implemented on a hardware processor, or (iii) acombination of (i) and (ii); any of (i)-(iii) implement the specifictechniques set forth herein.

Techniques of the present invention can provide substantial beneficialtechnical effects. Embodiments of the present invention can be applied,for example, to monitor tissue development to detect abnormaldevelopment. Exemplary output parameters of a system configuredaccording to an embodiment of the present invention, such as celldivision time, death time, migration rate etc, can enable biologists tomodel cell behavior and develop techniques to fight disease, such asCancer.

These and other features and advantages of the present invention willbecome apparent from the following detailed description of illustrativeembodiments thereof, which is to be read in connection with theaccompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Preferred embodiments of the present invention will be described belowin more detail, with reference to the accompanying drawings:

FIG. 1 is an input image of fluorescent Chinese hamster (Cricetulusgriseus) ovarian cell according to an embodiment of the presentinvention;

FIG. 2 is an exemplary cell segmentation of FIG. 1 using a watershedsegmentation method;

FIG. 3 is a ground truth segmentation of FIG. 1;

FIG. 4 is a flow diagram of a method of determining cell migrationparameters according to an embodiment of the present invention;

FIG. 5 is a flow diagram of a method of assessing risk according to anembodiment of the present invention;

FIG. 6 is a flow diagram of a method of producing temporal celltrajectories according to an embodiment of the present invention;

FIG. 7 is an illustration of a tracks detected in a sequence of imagesaccording to an embodiment of the present invention; and

FIG. 8 is a block diagram depicting an exemplary computer systemembodying a method for producing temporal cell trajectories andperforming a risk assessment, according to an exemplary embodiment ofthe present invention.

DETAILED DESCRIPTION

According to an exemplary embodiment of the present invention, a modelof temporal cell behavior is trained and used in performing an atomicrisk assessment. The model is applicable to predict a critical point, atime when the cell behavior will be risky, enabling improved decisionmaking in a patient's treatment plan.

Tracking multiple cells in time-lapse microscopy images (hereinafter,image data) is a challenging task. For example, cells tend to changeappearance and location, similar nearby cells need to be tracked toavoid misapprehension, and many cells are prone to missed detection orocclusion. For example, given an input image (see FIG. 1, 100), atypical segmentation method such as a watershed segmentation methodresults in a segmentation result that may exhibit a misidentificationproblem (see FIG. 2, 200) when compared to a ground truth (see FIG. 3,300).

Cell image analysis methods typically assume that the appearance of acell does not change over time, and do not consider dynamic information,such as cell velocity, at all. However, during the cell cycle, both cellappearance and cell velocity change over the time.

According to an exemplary embodiment of the present invention, a modelof temporal cell behavior is trained (see FIG. 4) and used in generatingone or more risk assessments (see FIG. 5).

According to an embodiment of the present invention and referring toFIG. 6, a training and risk assessment method 600 includes obtainingimage data 601, detecting cells in the image data 602, modeling thecells over time to generate a tree structure 603 (see FIG. 7), detectingcell tracks 604 (see also FIG. 4, data structure 406), and performing arisk assessment 605 (see FIG. 5). According to an embodiment of thepresent invention, the risk assessment is based on high-level temporalfeatures extracted from the cell trajectories (see FIG. 4, block 407).

According to an embodiment of the present invention, the image data 601,including a plurality of image fames, is obtained over a period of time(e.g., a few hours). It should be understood that the image data can beobtained over different periods of time sufficient to track cell linage.For example, the period of time may be selected based on the life cycleof the cells under investigations.

Referring again to FIG. 4, according to an embodiment of the presentinvention a method of generating cell migration parameters 400 includesobtaining image data 401, a modeling phase 402. The modeling phase 402includes detecting cells in the obtained image data 403 (see also FIG.6, block 601), performing a global association of cells in consecutiveframes 404 and mitosis modeling of the cells 405.

According to an embodiment of the present invention, the image data 401is obtained by collecting the (live) cells of interest (e.g., by biopsyof a patient, from culture, etc.). These cells are prepared for imaging,e.g., by presentation in a tissue culture dish with an appropriate mediafor sustaining the cells and/or imaging. The prepare cells are thenimaged by using an imaging device, such as, a wide-field microscope.

According to an embodiment of the present invention, in the modelingphase 402, the cell detection 403 includes segmenting cells of interestin a foreground area of individual image frames from a background areaof the image frames. According to exemplary embodiments of the presentinvention, different cell image analysis methods can be used to performthe segmentation at block 403. In at least one embodiment of the presentinvention a convolutional network is used to detect cells in the imagedata using features cues learned at different layers of theconvolutional network and with varying resolutions and scales. In one ormore embodiments of the present invention a supervised machine learningsystem is used to performed the segmentation and detect cells, whereinthe supervised machine learning system is trained to recognize cells ofinterest.

According to an embodiment of the present invention, the modeling phase402 distinguishes different cell instances segmented in the image dataover time, tracking individual cells over time using a motion estimationusing both forward and backward searching. According to an embodiment ofthe present invention, during the modeling phase 402, the motion of acell is tracked by generating a tracklet, which records the position ofthe cell from one frame to the next. Each tracklet includes a head and atail, and new positions of cell (i.e., in a next subsequent frame) areadded to the tail of the tracklet.

According to an exemplary embodiment of the present invention, themodeling phase 402 includes a forward pass 404 in which a set of initialcell tracks are generated. According to an embodiment of the presentinvention, the global association 404 finds the same cell in differentimage frames of the image data. According to an embodiment of thepresent invention, the global association 404 performs a dataassociation in which similar cells are connected across time. Connectedcells are assigned a same ID during cell tracking (see FIG. 6, block604). More particularly, each initial cell track corresponds to aninitial estimate of cell movement. In the forward pass 404, each cell ina given image frame (i.e., at a certain time) is associated with onesuccessor cell in the subsequent frame. According to an embodiment ofthe present invention, the association of cells between two frames isdefined as a one-to-one relationship and is tracked using cellidentifications. That is, according to an embodiment of the presentinvention, cells of interest are assigned different identifications, andthe same cell in different frames has the same identification.

According to one or more embodiments of the present invention, in theforward pass 404, cells in sequential frames are associated byconsidering an overlap criterion. For example, if a cell (e.g., obtainedeither by the segmentation or detection) in a first frame overlaps morethan 60% of the area of a cell in a second frame, then the first andsecond cell are determined to be the same cell and an association isidentified. It can be assumed that the first frame and the second framehave a same absolute position, i.e., a field of view does not change.

According to one or more embodiments of the present invention, theforward pass 404 includes generating a bounding box to enclose the firstcell in a first frame. The bounding box can be generated to have an areaencompassing the cell, e.g., being a predefined percentage larger than asize of the cell, being a predetermined size selected for the cell underinvestigation (for example, and taking into consideration typicalmotions of the cell), etc. The bounding box is applied to a subsequentframe in a same position (i.e., a global or absolute position), which islikely to enclose the second cell. If the first cell and the second celloverlap (e.g., more than about 60%) in the bounding box in theconsecutive frames, then the first and second cell are determined to bethe same cell and an association is identified.

According to an embodiment of the present invention, the forward pass404 includes evaluating one or more feature similarity cues of cells inconsecutive frames. The feature similarity cues can include theappearance cue and motion cue. For example, a color histogram can becreated for a first cell in a first same (or bounding box) and for asecond cell in a second frame (or bounding box), and if the appearancerepresentation of a cell (i.e., the appearance cue) is sufficientsimilar, the cells are considered to be the same cell and areassociated. According to at least one embodiment of the presentinvention, a motion model is used in determining an association. Similarto the Kalman filter, the motion model is obtained by determining avelocity vector from a head location to a tail location (i.e., a head ofthe first cell tracklet to a head of second tracklet) of any two celltracklets. If the locations of the heads of the first cell tracklet andthe second cell tracklet are sufficiently close, then the cellscorresponding to the first cell tracklet and the second cell trackletare considered to be the same cell and an association is identified.

It should be understood that the forward pass 404 can includeapplications of the overlap criteria together with evaluations of one ormore feature similarity cues, which generates a cumulative likelihood ofa cell being the same cell in consecutive frames. Accordingly, in one ormore embodiments of the present invention, a cumulative likelihood isused in identifying associations between cells in consecutive frames.

According to an embodiment of the present invention, the modeling phase402 further includes mitosis modeling 405, performed to detect cellsplitting captured in the image data. According to an exemplaryembodiment of the present invention, mitosis modeling 405 is a backwardpass tracing cell motion back in time to model mitosis events. In themitosis modeling 405, previous positions of the cells of interest arepredicted (e.g., each cell in the given image frame is assigned to onepredecessor cell in a previous frame) based on the motion estimationfrom the forward pass 404. The mitosis modeling 405 detects a mothercell whenever two cell tracks claim a same cell in a previous frame withat least a threshold probability (e.g., 95% confidence). According to anembodiment of the present invention, the threshold probability isanalogous to the confidence of tracklet, which combines an analysis oftwo or more feature similarities scores (e.g., motion and appearancesimilarities). According to an embodiment of the present invention, themitosis modeling 405 includes modeling cell cycle duration, modeling arate of mitosis and labeling, automatically, a plurality of cells withdifferent identities.

According to an exemplary embodiment of the present invention, thesystem handles mitosis events by assuming that the mother cell will bedisappeared in a subsequent frame and assigning two new identificationsto two daughter cells in the subsequent frame, wherein the two daughtercells are detected as being produced by the mother cell in the mitosisevent. According to an exemplary embodiment of the present invention,occluded (miss-detect) cells are distinguished from the process ofmitosis using cell motion information produced by the global association404. For example, if two tracklets of different cells converge to a sameposition, potentially being segmented as only a single cell in one ormore intermediate frames, and subsequently separating on theirrespective tracklets, then this series of events is captured andidentified, and no mitosis event will be identified. During theocclusion a dummy node is inserted in the tree to capture the occlusionevent.

In view of the foregoing, according to an embodiment of the presentinvention three or more temporal frames are used to model the motion ofeach cell of interest (e.g., a given or current frame, a previous frameand a subsequent frame). According to an exemplary embodiment of thepresent invention, motion information is obtained by determining avelocity vector from a head location to a tail location (i.e., the headof a first track to the tail of a second track) of any two tracksdetermined for the cells segmented from the image data and associated inthe forward pass 404.

Due to the changes in cell size, shape and brightness during mitosis,using only appearance information in a forward pass may be insufficientto make accurate associations between the cells 404 for the mitosismodeling 405. For example, in the case of similar moving cells, it canbe difficult to distinguish cells only using the appearance cue; in thisscenario, the information obtained from the dynamic information (e.g.,movement detected using the forward pass 404 and backward pass 405)provides complementary information for a cell tracker to find correcttrajectories for the cells of interest.

According to an exemplary embodiment of the present invention, themodeling phase 402 outputs a data structure (406) tracking cells inspace over time (see also FIG. 7). The upper face 411 of the datastructure (406) represents a first frame, giving an initial position ofone or more cells. The data structure (406), thus includes a pluralityof slices from an initial frame (i.e., time), with each framecorresponding to a slice, such that the position of a cell isrepresented through time as a tracklet, e.g., 412.

According to an exemplary embodiment of the present invention,high-level cell information 407 is generated from the cell trackingresults 406. According to an exemplary embodiment of the presentinvention, high-level cell information 407 includes, for example, cellvelocity and direction 408, the rate of mitosis 409 and morphologicalchanges in the cells 410. This data can be useful for detecting abnormalcell (and/or tissue) development during a risk assessment (see FIG. 5).

Referring to the risk assessment 500, according to an exemplaryembodiment of the present invention, high-level temporal features 501are extracted from the previously obtained cell trajectories 407.According to an exemplary embodiment of the present invention, fornormal and abnormal cell migration cases, a Recurrent Neural Network(RNN) is trained 502 using these high-level temporal features. Accordingto an exemplary embodiment of the present invention, the system predictstemporal cell behavior via the temporal trained model and performs riskassessment atomically 503. The system uses the temporal data to predicta critical point, a time when the cell behavior will be risky, enablingaccurate decisions to be made in a patient's treatment plan.

According to an exemplary embodiment of the present invention, once acell tree lineage is generated by the tracking framework, the high-levelinformation is extracted 407 to perform the risk assessment 500.According to an exemplary embodiment of the present invention, thesystem uses the RNN 502 to model temporal cell behavior andautomatically perform a risk assessment 503. According to an exemplaryembodiment of the present invention, the threshold of the riskassessment depends on the cell type, body tissues and cancer type.Accordingly, various thresholds can be used for the risk assessment 503.According to an exemplary embodiment of the present invention, the riskassessment 503 is used in determining a status of tissue development504, such as in tracking the stages of a pre-cancerous tissue, and tomake determinations of normal 505 or abnormal tissue status 506. In atleast one exemplary embodiment of the present invention, the status oftissue development 504 detects an abnormal condition as the uncontrolledmigration (e.g., by computing a migration rate) of the cells, andgenerates a risk of a tumor spreading and or cancer progression at 506.

According to one or more embodiments of the present invention, theabnormal condition is determined upon detecting a high mitotic rate(mitoses/millimeter²) with a high number of cells dividing. The highmitotic rate is associated with a cancer that is more likely to grow andspread. For example, the mitotic rate is useful in determining a stageof thin melanomas. According to one or more embodiments of the presentinvention, the mitosis rate criteria is used to specify different stagesof cancer development and abnormalities of the cell growth. An exampleof a staging system for melanoma, and useful in conjunction with one ormore embodiments of the present invention, includes the American JointCommission on Cancer (AJCC) TNM system, which is based on tumorinformation (T describing a size of original tumor, N describing anumber of nearby lymph nodes, and M describing metastasis). For example,T1a: The melanoma is less than or equal to 1.0 millimeter thick (1.0millimeter= 1/25 of an inch), without ulceration and with a mitotic rateof less than 1/millimeter^(2.) T1b: The melanoma is less than or equalto 1.0 mm thick. It is ulcerated and/or the mitotic rate is equal to orgreater than 1/mm².

According to an exemplary embodiment of the present invention, thesystem addresses missed detected cells in the assignment step. Forexample, a dummy node is inserted in the tree representing an occludedcell or a mis-detected cell (i.e., a cell originally occluded orotherwise mis-detected in a prior frame.

Referring again to the modeling phase 402, and more particularly thegeneration of the data structure (406) tracking cells in space overtime, FIG. 7 illustrates the method, wherein dashed lines, e.g., 701show initial tracks hypotheses and solid lines, e.g., 702, representfinal cell tracks after pruning of the initial tracks. Preformed atblock 603 of FIG. 6, the pruning keeps those cell tracks that morelikely represent the real cell trajectories. According to an embodimentof the present invention, pruning can be performing using a thresholdfor confidence on the global association and/or mitosis modeling. Forexample, the system can prune tracks that have confidences less than athreshold (e.g., less than 75% confidence). According to at least oneembodiment, the pruning is performed after the global association 404.

According to one or more embodiments of the present invention, each celltracklet has a confidence, which is combination of feature similarityscores (e.g., motion and appearance similarities). During the assignmentbetween the cells of each time step, tracklets having a confidence isless than the threshold are not chosen and are pruned from the datastructure, since these candidate cells are more likely to be noise (e.g.false positives).

At block 604, feature cues are used to detect cell tracks in the imagedata over time. (This aligns with step 404 in FIG. 4) According to anembodiment of the present invention, the feature cues include appearancecues of the cells and motion information (cell dynamics). The appearancecues and motion information are used to model the similarities of anypairwise cells over time. According to an exemplary embodiment of thepresent invention, a set of cell track hypothesis is constructed. Eachcell track hypothesis is associated with a score. This score is weightedcombination of two scores: appearance similarity score and motionsimilarities score between the temporal cells. The more score, it meansthey (the cells) more likely belong to the same cell track.)

Recapitulation:

According to an embodiment of the present invention, a system isconfigured to track cells (FIG. 4) and generating risk assessments (FIG.5). According to an embodiment of the present invention, FIG. 6illustrates a method 600 including obtaining image data 601, detectingcells in the image data 602, performing a modeling of the cells overtime to generate a tree structure 603 (see FIG. 7), detecting celltracks 604 (see also FIG. 4, data structure 406), and performing riskassessment 605 (see FIG. 5). According to an embodiment of the presentinvention, the risk assessment is based on high-level temporal featuresextracted from the cell trajectories (see FIG. 4, block 407).

Elements of one or more embodiments of the invention can be implemented,at least in part, in the form of an apparatus including a memory and atleast one processor that is coupled to the memory and operative toperform exemplary method steps.

One or more embodiments can make use of software running on a generalpurpose computer or workstation (e.g., to implement steps 601-605 and/orto carry cell tracking and risk assessment). With reference to FIG. 8,such an implementation might employ, for example, a processor 801, amemory 802, and an input/output interface formed, for example, by adisplay 803 and an input device 804 (e.g., keyboard). The term“processor” as used herein is intended to include any processing device,such as, for example, one that includes a CPU (central processing unit)and/or other forms of processing circuitry. Further, the term“processor” may refer to more than one individual processor. The term“memory” is intended to include memory associated with a processor orCPU, such as, for example, RAM (random access memory), ROM (read onlymemory), a fixed memory device (for example, hard drive), a removablememory device (for example, diskette), a flash memory and the like. Inaddition, the phrase “input/output interface” as used herein, isintended to include, for example, one or more mechanisms for inputtingdata to the processing unit (for example, mouse), and one or moremechanisms for providing results associated with the processing unit(for example, printer). The processor 801, memory 802, and input/outputinterface such as display 803 and input device 804 can beinterconnected, for example, via bus 805 as part of a data processingunit 800. Suitable interconnections, for example via bus 805, can alsobe provided to a network interface 806, such as a network card, whichcan be provided to interface with a computer network; to a mediainterface 807, such as a diskette or CD-ROM drive, which can be providedto interface with media 808; and/or to a sensor interface 809, such asanalog-to-digital converter(s) or the like, which can be provided tointerface with sensors or items to be controlled; e.g., the references,antenna elements, phase shifters, amps, etc.

Interfaces 806, 807 and 809 are generally representative of a variety oftechniques for communicating with and controlling the various elementsdiscussed herein. For example, processor 801 can communicate with thesensors, antenna elements, etc. over a wired or wireless computernetwork or directly with cabling.

A suitable optimization module may be stored in persistent memory andloaded into volatile memory to configure processor 801 to carry out thetechniques described herein.

Accordingly, computer software including instructions or code forperforming the methodologies of the invention, as described herein, maybe stored in one or more of the associated memory devices (for example,ROM, fixed or removable memory) and, when ready to be utilized, loadedin part or in whole (for example, into RAM) and implemented by a CPU.Such software could include, but is not limited to, firmware, residentsoftware, microcode, and the like.

A data processing system suitable for storing and/or executing programcode will include at least one processor 801 coupled directly orindirectly to memory elements 802 through a system bus 805. The memoryelements can include local memory employed during actual implementationof the program code, bulk storage, and cache memories which providetemporary storage of at least some program code in order to reduce thenumber of times code must be retrieved from bulk storage duringimplementation.

Input/output or I/O devices (including but not limited to keyboards,displays, pointing devices, and the like) can be coupled to the systemeither directly (such as via bus 805) or through intervening I/Ocontrollers (omitted for clarity).

Network adapters such as network interface 805 may also be coupled tothe system to enable the data processing system to become coupled toother data processing systems or remote printers or storage devicesthrough intervening private or public networks. Modems, cable modem andEthernet cards are just a few of the currently available types ofnetwork adapters. In one or more embodiments, network interface 806and/or sensor interface 809 collect data and also send control signals.

Computer-human interfaces can be provided using, for example, a suitablegraphical user interface (GUI) wherein a server serves html out to abrowser on a user's client machine. Interfaces between software and/orhardware elements can employ hard-wired connections, networks, databaseprograms to retrieve parameters from persistent storage, applicationprogramming interfaces (APIs), shared data structures, and the like.

As used herein, including the claims, a “server” includes a physicaldata processing system (for example, system 800 as shown in FIG. 8)running a server program. It will be understood that such a physicalserver may or may not include a display and keyboard.

It should be noted that any of the methods described herein can includean additional step of providing a system comprising distinct softwaremodules embodied on a computer readable storage medium; the modules caninclude, for example, any or all of the elements depicted in the blockdiagrams or other figures and/or described herein. The method steps canthen be carried out using the distinct software modules and/orsub-modules of the system, as described above, executing on one or morehardware processors 802. Further, a computer program product can includea computer-readable storage medium with code adapted to be implementedto carry out one or more method steps described herein, including theprovision of the system with the distinct software modules.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

1. A method of tracking a cell through a plurality of images comprising: selecting at least one cell in at least one image obtained at a first time; generating a track of the at least one cell through a plurality of images, including the at least one image, obtained at different times using a forward tracking and a backward tracking; and generating a cell tree lineage of the at least one cell using the track.
 2. The method of claim 1, wherein generating the track of the cell through the plurality of images further comprises modeling a cell cycle duration of the cell.
 3. The method of claim 1, wherein generating the track of the cell through the plurality of images further comprises using the backward tracking in modeling a rate of mitosis of the cell.
 4. The method of claim 1, further comprising labeling, simultaneously, a plurality of cells, including the at least one cell, with different identities in the plurality of images.
 5. The method of claim 1, further comprising visualizing a progression of the cell over time through a graphical tree structure.
 6. The method of claim 1, further comprising building a tree of potential track hypotheses for the cell.
 7. The method of claim 6, further comprising inserting a dummy node in the tree to represent an occluded cell.
 8. The method of claim 1, further comprising analyzing an appearance cue of the cell accounting for a similarity of the cell to a pairwise cell over time.
 9. The method of claim 1, further comprising inferring high-level information from the track including at least one of a rate of mitosis of the cell and a velocity of the cell.
 10. A method of tracking a cell through a plurality of images comprising: selecting at least one cell in at least one image obtained at a first time; generating a track of the at least one cell through a plurality of images, including the at least one image, obtained at different times using a forward tracking; performing a backward tracking tracing the at least one cell back in time until detecting a mitosis event generating the at least one cell; modeling a cell cycle duration of the at least one cell using a time of the mitosis event as a starting time; modeling a rate of mitosis of the at least one cell using the backward tracking; labeling, simultaneously, a plurality of cells, including the at least one cell, with different identities in the plurality of images; and generating a cell tree lineage of the plurality of cells using the track.
 11. The method of claim 10, further comprising: determining a confidence for each of the tracks; and pruning at least one track having a confidence less than a threshold.
 12. The method of claim 11, wherein the pruning is performed prior to the backward tracking.
 13. The method of claim 11, wherein the confidence of each of the tracks is determined using a combination of feature similarity scores for the respective cell over time.
 14. The method of claim 10, further comprising: detecting an abnormal condition in the cell tree lineage; and determining a risk associated with the plurality of cells.
 15. The method of claim 10, wherein generating the cell tree linage comprises outputting a representation comprising a plurality of slices corresponding to the plurality of images arranged in order of time, wherein the presentation includes a visualization of a plurality of tracklets, each tracklet corresponding to a location of a given one of the plurality of cells over time. 16-20. (canceled) 